In the operation of oil wells, a critical issue is how to control the flow rate of oil such that revenue from the well is maximized. In long horizontal or multi-lateral wells, it may be advantageous to separately control rates in different parts of the well, e.g., to delay water incursions. Such control can be achieved through downhole flow control valves. The flow control valves (FCVs) are installed underground in wells to regulate the flow of crude oil. A typical example is shown in FIGS. 1A and 1B (prior art), where there are three horizontal boreholes in a well, with an FCV at the head of each borehole. In practice, operators closely monitor the geophysical properties of the well and dynamically adjust the FCVs such that maximum revenue from the well can be attained. Here we are concerned about expected revenue because the evolution of the geophysical nature of the well is a stochastic process, with grid, aquifer strength and oil-water contact as the uncertainty in the system. For a risk-neutral operator, the difference between the expected revenue of the well without FCVs and the expected revenue of the well with FCVs installed and optimally controlled with future measurements gives the value of the FCV itself. Our task is to find the optimal control strategy and hence the value of the FCVs.
There are two major obstacles before us, the curse of dimensionality and the non-Markovian property of the problem. To derive the optimal value, we model the downhole flow control problem as a dynamic programming problem and solve for the optimal control policy backwardly. In order to derive the maximum production, the operator has to be forward-looking in decision-making. The decision made in the current period will affect the measurements observed in the future and hence the decisions made in the future. The operator must therefore take future measurements and decisions into account when setting the valves in the current period. In other words, the operator learns from future information. The fundamental theory and standard computational technique for dynamic programming can be found in the literature. However, application of dynamic programming to real world problems is often hindered by the so-called curse of dimensionality, which means that the state space grows exponentially with the number of state variables. To get rid of this obstacle, various approximate methods have been proposed, and a detailed and comprehensive description can be found in the literature.
Another obstacle we face is the non-Markovian property of the problem. The payoff of the well is associated with the operation of valves and the successive measurements. It can be difficult/wrong to encode the payoff as a function of only current measurements, when the payoff depends on the system trajectory or history, and not on the current state alone. In other words, the payoff not only depends on the current measurements and valves setting, but also on previous measurements and valve settings. This non-Markovian property of the problem poses a major difficulty in the valuation problem, since previous actions enter into the problem as the states for later periods, exacerbating the curse of dimensionality effect. Theoretically, to exactly solve a non-Markovian problem, we need to enumerate all possible settings for FCVs and, under each possible setting, we generate the evolution of geophysical properties and revenue by simulation. After all possible results have been obtained, we can search for the optimal value using standard backward induction. While this method is straightforward and accurate, it is hardly feasible in reality due to its immense computational demand. Consider the case in FIG. 1. Even for a simplified case where the FCVs are adjusted only three times throughout the life of the well, it took us a couple of months to generate all those simulations on a small Linux cluster. In fact, if we use eight Eclipse grids, three time periods, four settings for each valve, fixed setting for one valve, two aquifer strength samples, three oil-water contact samples, we need a total of 8×4×(42)3×2×3=786,432 Eclipse simulations. If each simulation takes an average of four minutes, it would require 2184 days to complete all simulations on a single computer. Detailed discussion of the optimal policy in this three-period model is presented in Section 2.